Add Run Android LlamaDemo with QNN backend #16011

luffy-yu · 2025-12-01T03:07:58Z

Motivation

The current Android LlamaDemo only supports XNNPACK.
The QNN-backend Android Demo is missing, the qnn_llama_runner approach is provided though.

Summary

Add Guidance for building and running Android LlamaDemo with QNN-Backend
It's verified on Samsung S23 (SoC SM8550).

Test plan

This PR has been tested on LlamaDemo-Executorch-QNN.

The APK file and the pre-built model can be found in its README.

This PR primarily borrows doc from QNN_ANDROID_FIX_SUMMARY.

pytorch-bot · 2025-12-01T03:08:01Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16011

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 3865295 with merge base 12d17ef ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

meta-cla · 2025-12-01T03:08:03Z

Hi @luffy-yu!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

Copilot

Pull request overview

This PR adds comprehensive documentation for building and running the Android LlamaDemo application with Qualcomm's QNN backend support. Previously, the Android demo only supported XNNPACK, and this guide fills that gap by providing step-by-step instructions verified on Samsung S23 hardware.

Key changes:

Added complete build and deployment guide for QNN-enabled Android LlamaDemo
Included verification steps to confirm correct AAR and APK configuration
Documented common issues and their solutions for QNN Android deployment

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/source/backends-qualcomm.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/source/backends-qualcomm.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-01T03:13:22Z

docs/source/backends-qualcomm.md

+### Run [Android LlamaDemo](https://github.com/meta-pytorch/executorch-examples/tree/main/llm/android/LlamaDemo) with QNN backend
+


The external link points to 'meta-pytorch/executorch-examples' but the PR is in the 'pytorch/executorch' repository. Verify this is the correct repository reference for the LlamaDemo, as it may confuse users if the link is incorrect or if the demo has moved.

Suggested change

### Run [Android LlamaDemo](https://github.com/meta-pytorch/executorch-examples/tree/main/llm/android/LlamaDemo) with QNN backend

### Run [Android LlamaDemo](https://github.com/pytorch/executorch/tree/main/examples/llm/android/LlamaDemo) with QNN backend

> **Note:** If the LlamaDemo is not present in the `pytorch/executorch` repository, you can find it in the [meta-pytorch/executorch-examples](https://github.com/meta-pytorch/executorch-examples/tree/main/llm/android/LlamaDemo) repository.

Copilot · 2025-12-01T03:13:22Z

docs/source/backends-qualcomm.md

+# Build the AAR
+cd $EXECUTORCH_ROOT
+export BUILD_AAR_DIR=$EXECUTORCH_ROOT/aar-out
+./scripts/build_android_library.sh


The script invocation lacks context about required environment variables or prerequisites. Consider adding a note about any required environment setup (e.g., Android NDK path, QNN SDK path) before running this script, or reference existing documentation that covers these prerequisites.

Copilot · 2025-12-01T03:13:23Z

docs/source/backends-qualcomm.md

+adb push model.pte /data/local/tmp/llama
+adb push tokenizer.bin /data/local/tmp/llama


The documentation doesn't specify where to obtain 'model.pte' and 'tokenizer.bin'. Add a reference to instructions for generating or downloading these required files, especially since the QNN backend requires specific model compilation steps.

Copilot · 2025-12-01T03:13:23Z

docs/source/backends-qualcomm.md

+# Check for debug strings in the AAR
+unzip -p $DEMO_APP/app/libs/executorch.aar jni/arm64-v8a/libexecutorch.so | \
+  strings | grep "YOUR DEBUG INFO"


The placeholder 'YOUR DEBUG INFO' is ambiguous and doesn't provide actionable guidance. Either provide a concrete example of what debug string to search for, or explain what type of debug information users should add to their code for verification purposes.

Suggested change

# Check for debug strings in the AAR

unzip -p $DEMO_APP/app/libs/executorch.aar jni/arm64-v8a/libexecutorch.so | \

strings | grep "YOUR DEBUG INFO"

# Check for your custom debug string in the AAR.

# For example, if you added a debug string like "MyCustomDebugString" to your code,

unzip -p $DEMO_APP/app/libs/executorch.aar jni/arm64-v8a/libexecutorch.so | \

strings | grep "MyCustomDebugString"

# Replace "MyCustomDebugString" with the actual debug string you added to your code.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-01T03:14:36Z

docs/source/backends-qualcomm.md

+
+**Solution**: 
+1. Update `build.gradle.kts` with matching QNN runtime version
+


The note appears after the 'Before' label but should appear before showing any code examples. Moving it above line 395 would make the documentation flow more clearly and prevent confusion about which versions to use.

Suggested change

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-01T03:15:16Z

docs/source/backends-qualcomm.md

+unzip -l app/build/outputs/apk/debug/app-debug.apk | grep "libQnnHtp.so"
+```
+
+Expected size for QNN 2.37.0: ~2,465,440 bytes


[nitpick] The expected file size is overly specific and may vary slightly between builds or minor version updates. Consider changing this to a range (e.g., '~2.4-2.5 MB') to make the verification more resilient to minor variations.

Suggested change

Expected size for QNN 2.37.0: ~2,465,440 bytes

Expected size for QNN 2.37.0: ~2.4–2.5 MB (2,400,000–2,500,000 bytes)

Copilot · 2025-12-01T03:15:16Z

docs/source/backends-qualcomm.md

+**Solution**: 
+1. Update `build.gradle.kts` with matching QNN runtime version
+
+> **Note:** The version numbers below (`2.33.0` and `2.37.0`) are examples only. Please check for the latest compatible QNN runtime version or match your QNN SDK version to avoid API mismatches.


[nitpick] The note about version numbers being examples appears after mentioning 'version X' and 'version Y' in the previous text but before showing the actual version numbers. This note would be more effective if placed immediately after the code examples (after line 403) where the specific versions are shown, or reference should be made to it in the text above where X and Y are mentioned.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-01T03:16:28Z

docs/source/backends-qualcomm.md

+> **Note:** The version numbers below (`2.33.0` and `2.37.0`) are examples only. Please check for the latest compatible QNN runtime version or match your QNN SDK version to avoid API mismatches.
+**Before**:
+```kotlin
+implementation("com.qualcomm.qti:qnn-runtime:2.33.0")
+```
+
+**After**:
+```kotlin
+implementation("com.qualcomm.qti:qnn-runtime:2.37.0")
+```


The numbered list item '1.' is followed by a note and code examples, but there's no item '2.' at the same indentation level until line 405. The note and code examples should be indented or formatted as sub-items of item 1 to maintain proper list structure.

Suggested change

> **Note:** The version numbers below (`2.33.0` and `2.37.0`) are examples only. Please check for the latest compatible QNN runtime version or match your QNN SDK version to avoid API mismatches.

**Before**:

```kotlin

implementation("com.qualcomm.qti:qnn-runtime:2.33.0")

```

**After**:

```kotlin

implementation("com.qualcomm.qti:qnn-runtime:2.37.0")

```

> **Note:** The version numbers below (`2.33.0` and `2.37.0`) are examples only. Please check for the latest compatible QNN runtime version or match your QNN SDK version to avoid API mismatches.

**Before**:

```kotlin

implementation("com.qualcomm.qti:qnn-runtime:2.33.0")

```

**After**:

```kotlin

implementation("com.qualcomm.qti:qnn-runtime:2.37.0")

```

cccclai · 2025-12-02T20:25:01Z

@haowhsu-quic can I get someone to review this PR? Or maybe suggestions regarding how to structure this

haowhsu-quic · 2025-12-03T03:22:53Z

docs/source/backends-qualcomm.md

+1. Update `build.gradle.kts` with matching QNN runtime version
+
+> **Note:** The version numbers below (`2.33.0` and `2.37.0`) are examples only. Please check for the latest compatible QNN runtime version or match your QNN SDK version to avoid API mismatches.
+**Before**:


This one is merged into quotation above.

haowhsu-quic · 2025-12-03T03:23:01Z

docs/source/backends-qualcomm.md

+
+##### Issue 1: Error 18 (InvalidArgument)
+
+**Cause**: Wrong parameter order in Runner constructor or missing QNN config


It will be more readable if indentation could be applied to cause, symptoms and solution.

github-actions · 2025-12-03T03:45:26Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-03T03:51:27Z

docs/source/backends-qualcomm.md

+***Step 5***: Push model
+
+```bash
+adb shell mkdir -p /data/local/tmp/llama
+adb push model.pte /data/local/tmp/llama
+adb push tokenizer.bin /data/local/tmp/llama
+```


Step 5 references model.pte and tokenizer.bin without explaining where these files come from or providing a reference to model generation instructions. Consider adding a note before Step 5 that links to the model export documentation (e.g., "See How to Support a Custom Model in HTP Backend for model generation instructions" or pointing to the Llama model export process specifically).

Copilot · 2025-12-03T03:51:27Z

docs/source/backends-qualcomm.md

+***Step 1***: Rebuild ExecuTorch AAR
+
+```bash
+# Build the AAR
+cd $EXECUTORCH_ROOT
+export BUILD_AAR_DIR=$EXECUTORCH_ROOT/aar-out
+./scripts/build_android_library.sh
+```
+
+***Step 2***: Copy AAR to Android Project
+
+```bash
+cp $EXECUTORCH_ROOT/aar-out/executorch.aar \
+   $DEMO_APP/app/libs/executorch.aar
+```
+
+***Step 3***: Build Android APK
+
+```bash
+cd $DEMO_APP
+./gradlew clean assembleDebug -PuseLocalAar=true
+```
+
+***Step 4***: Install on Device
+
+```bash
+adb install -r app/build/outputs/apk/debug/app-debug.apk
+```
+
+***Step 5***: Push model
+
+```bash
+adb shell mkdir -p /data/local/tmp/llama
+adb push model.pte /data/local/tmp/llama
+adb push tokenizer.bin /data/local/tmp/llama
+```
+
+***Step 6***: Run the Llama Demo


The step formatting is inconsistent with the existing documentation style. Steps 1-6 use ***Step X***: (with a colon), but the existing document uses ***Step X***. (with a period) as seen in lines 230 and 248. Please update the formatting to match the existing style by replacing colons with periods after the step numbers.

haowhsu-quic

LGTM, thank you.

Add Run Android LlamaDemo with QNN backend

29f233a

Copilot AI review requested due to automatic review settings December 1, 2025 03:07

luffy-yu requested a review from mergennachin as a code owner December 1, 2025 03:07

Copilot AI reviewed Dec 1, 2025

View reviewed changes

docs/source/backends-qualcomm.md Outdated Show resolved Hide resolved

docs/source/backends-qualcomm.md Outdated Show resolved Hide resolved

docs/source/backends-qualcomm.md Show resolved Hide resolved

Fix ***Step 6***: Run the Llama Demo

2c39ea1

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings December 1, 2025 03:11

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 1, 2025

Copilot AI reviewed Dec 1, 2025

View reviewed changes

docs/source/backends-qualcomm.md Outdated Show resolved Hide resolved

docs/source/backends-qualcomm.md Outdated Show resolved Hide resolved

Update $DEMO_APP definition

5ba7196

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings December 1, 2025 03:12

Copilot AI reviewed Dec 1, 2025

View reviewed changes

Update version numbers example

c67fca4

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings December 1, 2025 03:14

Copilot AI reviewed Dec 1, 2025

View reviewed changes

Update grep "QNN"

975c371

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings December 1, 2025 03:14

Copilot AI reviewed Dec 1, 2025

View reviewed changes

Replace X.XX.X with 2.33.0

3731048

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings December 1, 2025 03:16

Copilot AI reviewed Dec 1, 2025

View reviewed changes

luffy-yu mentioned this pull request Dec 1, 2025

Executorch with QNN AI Engine Backend #15387

Open

cccclai requested review from chenweng-quic, haowhsu-quic, shewu-quic and winskuo-quic December 1, 2025 05:44

haowhsu-quic reviewed Dec 3, 2025

View reviewed changes

Copilot AI review requested due to automatic review settings December 3, 2025 03:46

luffy-yu force-pushed the main branch from a925ab0 to 3865295 Compare December 3, 2025 03:46

Copilot started reviewing on behalf of luffy-yu December 3, 2025 03:47 View session

Format with indentations

3865295

Copilot finished reviewing on behalf of luffy-yu December 3, 2025 03:49

Copilot AI reviewed Dec 3, 2025

View reviewed changes

haowhsu-quic approved these changes Dec 3, 2025

View reviewed changes

cccclai approved these changes Dec 4, 2025

View reviewed changes

cccclai merged commit 5b00d91 into pytorch:main Dec 4, 2025
146 of 147 checks passed

		### Run [Android LlamaDemo](https://github.com/meta-pytorch/executorch-examples/tree/main/llm/android/LlamaDemo) with QNN backend

-### Run [Android LlamaDemo](https://github.com/meta-pytorch/executorch-examples/tree/main/llm/android/LlamaDemo) with QNN backend
+### Run [Android LlamaDemo](https://github.com/pytorch/executorch/tree/main/examples/llm/android/LlamaDemo) with QNN backend
+> **Note:** If the LlamaDemo is not present in the `pytorch/executorch` repository, you can find it in the [meta-pytorch/executorch-examples](https://github.com/meta-pytorch/executorch-examples/tree/main/llm/android/LlamaDemo) repository.

		adb push model.pte /data/local/tmp/llama
		adb push tokenizer.bin /data/local/tmp/llama


		Solution:
		1. Update `build.gradle.kts` with matching QNN runtime version

	Expected size for QNN 2.37.0: ~2,465,440 bytes
	Expected size for QNN 2.37.0: ~2.4–2.5 MB (2,400,000–2,500,000 bytes)


		##### Issue 1: Error 18 (InvalidArgument)

		Cause: Wrong parameter order in Runner constructor or missing QNN config

Add Run Android LlamaDemo with QNN backend #16011

Add Run Android LlamaDemo with QNN backend #16011

Conversation

luffy-yu commented Dec 1, 2025

Motivation

Summary

Test plan

Uh oh!

pytorch-bot bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16011

✅ You can merge normally! (1 Unrelated Failure)

Uh oh!

meta-cla bot commented Dec 1, 2025

Action Required

Process

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

cccclai commented Dec 2, 2025

Uh oh!

haowhsu-quic Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

luffy-yu Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

haowhsu-quic Dec 3, 2025

pytorch-bot bot commented Dec 1, 2025 •

edited

Loading

This PR needs a `release notes:` label